System and method for stereoscopic photography

ABSTRACT

A stereoscopic photography is provided, which includes a first image sensor, a second image sensor, a synchronization module, a combination module, a calibration module, and a stereo matching module. The first image sensor is utilized to capture a first image. The second image sensor is utilized to capture a second image. The calibration module is utilized to perform an image calibration and a horizontal rectification for the first image and the second image. The stereo matching module is utilized to calculate a distance difference between a reference pixel of the first image and a corresponding pixel of the second image. The stereo matching module compares costs of multiple pixels relative to the reference pixel in a pixel row of the second image in an interval manner. A method for performing stereoscopic photography and a stereo matching method are also provided.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority from Taiwan Patent Application No.103110908, filed in the Taiwan Patent Office on Mar. 24, 2014, entitled“SYSTEM AND METHOD FOR STEREOSCOPIC PHOTOGRAPHY,” and incorporates theTaiwan patent application in its entirety by reference.

TECHNICAL FIELD OF THE INVENTION

The present invention relates to a stereoscopic photography system and amethod thereof, and in particular to a stereoscopic photography systemand a method which are capable of reducing the amount of computation.

BACKGROUND OF THE INVENTION

In recent years, with the development of three-dimensional (3D) displaytechnology, processing of stereoscopic images has been increasinglyimportant. In general, the stereoscopic images can be formed in thefollowing ways: for example, using a depth camera that can obtain depthinformation to photograph, or using dual cameras which can simulatehuman binocular vision to photograph, and then performing an appropriateimage processing for two-dimensional (2D) images to obtain thestereoscopic images.

Stereoscopic image herein means that the objects in the image havedifferent visual depths in addition to the usual two-dimensional images.The technique of converting the 2D images into the stereoscopic imagesis called stereo matching. Stereo matching means a process ofphotographing two or more images of a certain scene, estimating a 3Dmodel of the scene by accurately finding matching pixels between theimages, and converting 2D positions of the matching pixels into 3Ddepths.

In the computing technique of conventional stereo matching, one of thetwo images which are respectively captured by two cameras usually servesas a reference image, and the other serves as a target image. Then, adisparity map of the target image relative to the reference image isoutput. The disparity of each pixel is inversely proportional to thedistance of a photographed object. Therefore, the disparity map can beutilized to depict the 3D depths of the captured image.

However, each of the pixels in the reference image is required tocalculate the disparity thereof, and the algorithm of the conventionalstereo matching is very complicated, so there is a great amount ofcomputation. Thus, under the restriction of the current semiconductortechnology, the stereoscopic photography techniques using a dual cameraare still in a developmental stage, it is difficult to reach acommercialization stage.

SUMMARY OF THE INVENTION

Accordingly, an objective of the present invention is to provide astereoscopic photography system which provides a specified structure toachieve stereoscopic photography and is able to reduce the amount ofcomputations needed for stereo matching, so that the stereoscopicphotography system can be commercialized.

Another objective of the present invention is to provide a method forperforming stereoscopic photography which employs an interval manner tocalculate the disparities for reducing the amount of the computation,thereby solving the problem of the current semiconductor restriction.

Yet another objective of the present invention is to provide a stereomatching method for the stereoscopic photography system, which employsan interval manner to calculate the disparities for reducing the amountof computations, thereby realizing commercialization.

To achieve the foregoing objectives, according to an aspect of thepresent invention, the stereoscopic photography system provided in thepresent invention includes a first image sensor, a second image sensor,a synchronization module, a combination module, a calibration module,and a stereo matching module. The first image sensor is utilized tocapture a first image for generating a first data stream. The secondimage sensor is disposed apart from the first image sensor by ahorizontal distance and is utilized to capture a second image forgenerating a second data stream. The synchronization module iselectrically coupled to the first image sensor and the second imagesensor and is utilized to synchronize the first data stream and thesecond data stream and utilized to synchronize auto-exposure andauto-white balance parameters of the first image sensor and the secondimage sensor. The combination module is utilized to combine the firstdata stream and the second data stream for outputting a combined imagedata. The combined image data includes the first image and the secondimage. The calibration module is utilized to perform an imagecalibration and a horizontal rectification for the first image and thesecond image in the combined image data. The stereo matching module isutilized to calculate a distance difference between a reference pixel ofthe first image and a corresponding pixel of the second image. Thestereo matching module compares costs of a plurality of pixels relativeto the reference pixel in a pixel row of the second image in an intervalmanner.

In one preferred embodiment, the interval manner is an interval of apredetermined number of pixels. Specifically, the predetermined numberis a positive integer.

In one preferred embodiment, the costs are brightness differences.

In one preferred embodiment, the stereo matching module further includesa sub-pixel interpolation unit which is utilized to accurately computethe distance differences, so that an error of the distance differencesis less than 0.1 pixels.

In one preferred embodiment, the pixels are positioned in a searchinterval of the pixel row.

In one preferred embodiment, the synchronization module provides a clockcontrolling signal for the first image sensor and the second imagesensor. Moreover, the first data stream and the second data stream havesynchronized data line signals.

In one preferred embodiment, the stereo matching module is implementedby software, hardware, firmware, or a combination thereof.

To achieve the foregoing objectives, according to another aspect of thepresent invention, the method for performing stereoscopic photographyprovided in the present invention includes the steps of: capturing afirst image and a second image respectively by a first image sensor anda second image sensor for generating a first data stream and a seconddata stream; synchronizing the first data stream and the second datastream, and synchronizing auto-exposure and auto-white balanceparameters of the first image sensor and the second image sensor;combining the first data stream and the second data stream foroutputting a combined image data which comprises the first image and thesecond image; performing an image calibration and a horizontalrectification for the first image and the second image in the combinedimage data; and calculating a distance difference between a referencepixel of the first image and a corresponding pixel of the second image,wherein calculating the distance difference further comprises the stepof comparing costs of a plurality of pixels relative to the referencepixel in a pixel row of the second image in an interval manner.

In one preferred embodiment, the interval manner is an interval of apredetermined number of pixels. Specifically, the predetermined numberis a positive integer.

In one preferred embodiment, the costs are brightness differences.

In one preferred embodiment, calculating the distance difference furtherincludes the step of performing a sub-pixel interpolation for accuratelycomputing the distance difference, so that an error of the distancedifferences is less than 0.1 pixels.

In one preferred embodiment, the pixels are positioned in a searchinterval of the pixel row.

In one preferred embodiment, the step of calculating the distancedifference is implemented by software, hardware, firmware, or acombination thereof.

To achieve the foregoing objectives, according to yet another aspect ofthe present invention, the present invention provides a stereo matchingmethod for the stereoscopic photography system. The stereoscopicphotography system includes a first image sensor for capturing a firstimage and a second image sensor for capturing a second image. The methodincludes: calculating a distance difference between a reference pixel ofthe first image and a corresponding pixel of the second image, whereincalculating the distance difference further comprises the step ofcomparing costs of a plurality of pixels relative to the reference pixelin a pixel row of the second image in an interval manner.

In one preferred embodiment, the interval manner is an interval of apredetermined number of pixels, and the predetermined number is apositive integer.

In comparison with the prior art, the stereo matching module employed inthe present invention not only reduces the amount of disparitycomputations in the interval manner, but also improves the accuracy ofcalculating the 3D depths by means of sub-pixel interpolation. That isto say, the stereoscopic photography system and method of the presentinvention not only reduce the amount of computations by the intervalmanner, but also improves the accuracy of the 3D depths due to sub-pixelinterpolation.

It is to be understood that both the foregoing general description andthe following detailed description of the present invention areexemplary and explanatory and are intended to provide furtherexplanation of the invention as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a functional block diagram illustrating a stereoscopicphotography system according to one preferred embodiment of the presentinvention;

FIG. 2 is a timing chart schematically illustrating a first data streamand a second data stream;

FIG. 3 is a schematic drawing illustrating operation of a combinationmodule;

FIG. 4 is a schematic drawing illustrating operation of a stereomatching module;

FIG. 5 is a schematic drawing illustrating a first image and a secondimage;

FIG. 6 is a schematic drawing illustrating computation of a distancedifference;

FIG. 7 depicts a schematic drawing illustrating a cost functionaccording to the embodiment of the present invention; and

FIG. 8 is a flow chart illustrating a method for performing stereoscopicphotography according to one preferred embodiment of the presentinvention.

DETAILED DESCRIPTION OF THE INVENTION

The present invention will now be described in detail with reference toa few preferred embodiments thereof as illustrated in the accompanyingdrawings. The same reference numerals refer to the same parts or likeparts throughout the various figures.

FIG. 1 is a functional block diagram illustrating a stereoscopicphotography system according to one preferred embodiment of the presentinvention. The stereoscopic photography system 10 of the embodimentincludes a first image sensor 120, a second image sensor 140, asynchronization module 150, a combination module 160, an image signalprocessing module 170, a calibration module 180, and a stereo matchingmodule 190.

Specifically, The first image sensor 120 is utilized to capture a firstimage for generating a first data stream 220. The second image sensor140 is disposed apart from the first image sensor 120 by a horizontaldistance and is utilized to capture a second image for generating asecond data stream 240. Preferably, the horizontal distance is between 4and 8 centimeters. Specifically, the first image sensor 120 can be aleft camera for capturing a left image; the second image sensor 140 canbe a right camera for capturing a right image. Furthermore, the firstimage sensor 120 and the second image sensor 140 of the embodiment areimplemented by two RGB cameras, such as CMOS or CCD cameras, and the twoRGB cameras have the same or similar properties (e.g., resolution).However, the present invention is not restricted thereto.

The first image sensor 120 and second image sensor 140 preferably aredisposed on a circuit board (not shown). The synchronization module 150,the combination module 160, and the image signal processing module 170can be configured on the circuit board. The above-mentionedsynchronization module 150, the combination module 160 and the imagesignal processing module 170 can respectively be a chip, or they can beintegrated into a System-on-a-Chip (SoC) for reducing the size and costthereof.

Referring to FIG. 1 and FIG. 2, FIG. 2 is a timing chart schematicallyillustrating the first data stream 220 and the second data stream 240.The synchronization module 150 is electrically coupled to the firstimage sensor 120 and the second image sensor 140, and it is utilized tosynchronize the first data stream 220 and the second data stream 240. Asshown in FIG. 2, the first data stream 220 has a plurality of data linesignals 222(1)˜222(N). Each of the data line signals 222 includes theamount of information in each of the data lines, and these data linesform the data of a frame of the first image captured by the first imagesensor 120. Similarly, the second data stream 240 has a plurality ofdata line signals 242(1)˜242(N). Each of the data line signals 242includes the amount of information in each of the data lines, and thesedata lines form the data of the frame of the second image captured bythe second image sensor 140.

The synchronization module 150 provides a clock controlling signal 310for the first image sensor 120 and the second image sensor 140 tocontrol the timing sequences of the first data stream 220 and the seconddata stream 240 such that the first data stream 220 and the second datastream 240 have synchronized data line signals 222, 242 and a verticalsync signal (shown as dashed lines). The vertical sync signal means thatthe timing sequences of the first data stream 220 output a first frameF1 and the second data stream 240 output a second frame F2 aresynchronous.

On the other hand, the synchronization module 150 is capable ofcommunicating with the image signal processing module 170 forsynchronizing auto-exposure and auto-white balance parameters of thefirst image sensor 120 and the second image sensor 140. Accordingly, thebrightness and color of the first image and the second imagesimultaneously captured by both the first image sensor 120 and thesecond image sensor 140 are the same, in order to facilitate subsequentimage processes.

Referring to FIG. 1 and FIG. 3, FIG. 3 is a schematic drawingillustrating the operation of the combination module. The combinationmodule 160 is utilized to combine the first data stream 220 and thesecond data stream 240 for outputting a combined image data 260. Thecombined image data 260 includes the first image and the second image.Preferably, the area of the frame that is composed of the combined imagedata 260 is twice the area of the frame of the first image or the secondimage.

Subsequently, as shown in FIG. 1, the combined image data 260 is sent tothe image signal processing module 170 for image processing (such ascolor processing) and then is provided for the calibration module 180.More specifically, the combination module 160 combines the first datastream 220 and the second data stream 240 of the first image sensor 120and the second image sensor 140, so just one Image Signal Processor(ISP) needs to be employed in the processing.

As shown in FIG. 1, the calibration module 180 is electrically coupledto the image signal processing module 170. The calibration module 180 isutilized to perform an image calibration and a horizontal rectificationfor the first image and the second image in the combined image data 260.Image calibration herein is used for correcting the distortion of thefirst image and the second image or the distortion caused by lenses. Forexample, it is capable of adjustment according to factory parameters ofthe lenses. Besides, the horizontal rectification is used for aligningthe levels of the first image and the second image, so that the objectcaptured by both can be located at the same height (i.e., Y coordinate)on the first image and the second image.

Referring to FIG. 4 and FIG. 5, FIG. 4 is a schematic drawingillustrating the operation of the stereo matching module; FIG. 5 is aschematic drawing illustrating the first image and the second image. Thestereo matching module 190 is utilized to compute a disparity map 500 ofa first image 420 relative to a second image 440. Specifically,referring to FIG. 5, the stereo matching module 190 computes a distancedifference (i.e., disparity) between a reference pixel P_(L) in thefirst image 420 and a corresponding pixel P_(R) in the second image. Thecorresponding pixel P_(R) herein represents the pixel indicating theposition of the object, which is represented by the reference pixelP_(L) on the first image 420, on the second image 440. Moreover, sincethe horizontal rectification has been performed on the first image 420relative to the second image 440, the reference pixel P_(L) and thecorresponding pixel P_(R) are located at the same horizontal line of thefirst image 420 and the second image 440, and the horizontal line isalso known as an epipolar line 450. It can be seen from the foregoingthat the distance difference is also a horizontal distance difference,that is, the difference between a coordinate X_(L) of the referencepixel P_(L) on the first image 420 and a coordinate X_(R) of thecorresponding pixel P_(R) on the second image 440. In these coordinates,the centers (dashed line) of the first image sensor 120 and second imagesensor 140 serve as the origins, positive direction toward the right andnegative direction toward left. Therefore, the distance difference(disparity) is X_(R)−X_(L).

The method for computing the distance difference will be discussed indetail in the following sections. Referring to FIG. 6, FIG. 6 is aschematic drawing illustrating the computation of the distancedifference. Computing the distance difference by the stereo matchingmodule 190 further includes comparing costs of a plurality of pixelsrelative to the reference pixel P_(L) in a pixel row 460 (epipolar line450) of the second image 440 in an interval manner. Specifically, thecosts represent correlations between the pixels P and the referencepixel P_(L). For example, the correlations are brightness differences orcolor differences between the pixels P and the reference pixel P_(L). Onthe other hand, the interval manner is an interval of a predeterminednumber of the pixels P, in which the predetermined number is a positiveinteger. For example, if the predetermined number is 1, the amount ofcomputation can be reduced in the interval of one pixel P, as shown inFIG. 6. It is understandable that the amount of computations can behalved when the predetermined number is 1.

However, the present invention is not limited thereto. For example, thepredetermined number can be 2, 3, 4, 5, or other positive integers.Besides, every time the interval may not be a fixed value. It is worthmentioning that the computation for the selected pixels P on the entirepixel row 460 doesn't need to be carried out. It is only necessary toperform the comparison in the interval manner within a search interval Rbeginning at the coordinate X_(L) of the reference pixel P_(L) on thefirst image 420, thereby further reducing the amount of computations.

FIG. 7 depicts a schematic drawing illustrating a cost functionaccording to the embodiment of the present invention, where thehorizontal axis represents the brightness difference, and the verticalaxis represents the disparity (measured in pixels). After the count inthe interval manner by the interval of one pixel P as shown in FIG. 6, aplurality of data points D can be obtained. As shown in FIG. 4, thestereo matching module 190 further includes a sub-pixel interpolationunit 195. The sub-pixel interpolation unit is utilized to accuratelycompute the distance differences. Specifically, by these data points D,the stereo matching module 190 can obtain a cost function f (x) througha mathematical operation, as shown in FIG. 7. The sub-pixelinterpolation unit 195 can interpolate the cost function f (x) to obtainthe lowest point (triangles) of the cost function f (x), and thus obtainan accurate disparity so that an error of the distance differences isless than 0.1 pixels. That is to say, the distance difference calculatedby the sub-pixel interpolation unit 195 can have a higher resolution,thereby improving the matching accuracy.

After the stereo matching module 190 calculates the disparity of eachpixel P, the disparity map 500 can be acquired. Accordingly, the depthof objects is inversely proportional to the disparity and isproportional to the product of a focal length and a horizontal distancebetween the second image sensor 140 and the first image sensor 120.Therefore, the depth of objects can be derived from the disparity map500.

It is worth mentioning that the stereo matching module 190 isimplemented by software, hardware, firmware, or a combination thereof.Preferably, the calibration module 180 and the stereo matching module190 can be implemented by software. Moreover, the synchronization module150, the combination module 160 and the image signal processing module170 can be implemented by a microprocessor, one or moreapplication-specific integrated circuit (ASIC), one or morefield-programmable gate array (FPGA), or any combination thereof.

A stereoscopic photography method of the stereoscopic photography system10 of the embodiment will be explained in the following sections.Referring to FIG. 1 and FIG. 8, FIG. 8 is a flow chart illustrating amethod for performing stereoscopic photography according to onepreferred embodiment of the present invention. The method for performingthe stereoscopic photography according to the embodiment is used for theabove-mentioned stereoscopic photography system 10, and the descriptionsof the following elements have been explained above, so it is notnecessary to go into detail herein.

The method for performing the stereoscopic photography according to theembodiment begins with step S10. In step S10, the first image sensor 120and the second image sensor 140 respectively capture a first image and asecond image for generating a first data stream 220 and a second datastream 240, and then execution resumes at step S20.

In step S20, the synchronization module 150 synchronizes the first datastream 220 and the second data stream 240 and synchronizes auto-exposureand auto-white balance parameters of the first image sensor 120 and thesecond image sensor 140, and then execution resumes in step S30.

In step S30, the combination module 160 combines the first data stream220 and the second data stream 240 for outputting a combined image data260, and then execution resumes in step S40. The combined image data 260includes the first image and the second image.

In step S40, the calibration module 180 performs an image calibrationand a horizontal rectification for the first image and the second imagein the combined image data 260, and then execution resumes in step S50.

In step S50, the stereo matching module calculates a distance differencebetween a reference pixel P_(L) of the first image 420 and acorresponding pixel P_(R) of the second image 440, The step ofcalculating the distance difference further includes comparing costs ofa plurality of pixels P relative to the reference pixel P_(L) in a pixelrow 460 of the second image 440 in an interval manner. It is worthmentioning that the interval manner is an interval of a predeterminednumber of the pixels P. Specifically, the predetermined number is apositive integer.

Specifically, calculating the distance difference in step S50specifically includes the step of performing a sub-pixel interpolationfor accurately computing the distance difference, so that an error ofthe distance differences is lower than 0.1 pixels. Similarly,calculating the distance difference in step S50 is implemented bysoftware, hardware, firmware, or a combination thereof.

Similarly, the stereo matching method of the stereoscopic photographysystem 10 of the embodiment will be explained in detail below. Referringto FIG. 1 and FIG. 5, the stereoscopic photography system 10 includes afirst image sensor 120 for capturing a first image and a second imagesensor 140 for capturing a second image. The stereo matching methodincludes calculating a distance difference between a reference pixelP_(L) of the first image 420 and a corresponding pixel P_(R) of thesecond image 440. The step of calculating the distance differencefurther includes comparing costs of a plurality of pixels P relative tothe reference pixel P_(L) in a pixel row 460 of the second image 440 inan interval manner. More specifically, the interval manner is aninterval of a predetermined number of pixels, and the predeterminednumber is a positive integer.

The steps of a method or algorithm described in connection with theembodiments disclosed herein may be embodied directly in hardware, in asoftware module executed by a processor, or in a combination of the two.A software module may reside in random access memory (RAM), flashmemory, read-only memory (ROM), programmable read-only memory (PROM),erasable programmable read-only memory (EPROM), electrically erasableprogrammable read-only memory (EEPROM), registers, hard disk, aremovable disk, a compact disk read-only memory (CD-ROM), or any otherform of storage medium known in the art.

In summary, the stereo matching module 190 employed in the presentinvention not only decreases the amount of disparity computations in theinterval manner but also improves the accuracy of calculating the 3Ddepths by means of the sub-pixel interpolation. That is to say, thestereoscopic photography system and method of the present invention notonly reduce the amount of computation by the interval manner, but alsoimprove the accuracy of the 3D depths due to the sub-pixelinterpolation.

While the preferred embodiments of the present invention have beenillustrated and described in detail, various modifications andalterations can be made by persons skilled in this art. The embodimentof the present invention is therefore described in an illustrative butnot restrictive sense.

What is claimed is:
 1. A stereoscopic photography system, comprising: afirst image sensor utilized to capture a first image for generating afirst data stream; a second image sensor disposed apart from the firstimage sensor by a horizontal distance, utilized to capture a secondimage for generating a second data stream; a synchronization moduleelectrically coupled to the first image sensor and second image sensor,utilized to synchronize the first data stream and the second data streamand utilized to synchronize auto-exposure and auto-white balanceparameters of the first image sensor and the second image sensor; acombination module utilized to combine the first data stream and thesecond data stream for outputting a combined image data which comprisesthe first image and the second image; a calibration module utilized toperform an image calibration and a horizontal rectification for thefirst image and the second image in the combined image data; and astereo matching module utilized to calculate a distance differencebetween a reference pixel of the first image and a corresponding pixelof the second image, wherein the stereo matching module compares costsof a plurality of pixels relative to the reference pixel in a pixel rowof the second image in an interval manner.
 2. The stereoscopicphotography system of claim 1, wherein the interval manner is aninterval of a predetermined number of the pixels.
 3. The stereoscopicphotography system of claim 2, wherein the predetermined number is apositive integer.
 4. The stereoscopic photography system of claim 1,wherein the costs are brightness differences.
 5. The stereoscopicphotography system of claim 1, wherein the stereo matching modulefurther comprises a sub-pixel interpolation unit which is utilized tocompute the distance differences.
 6. The stereoscopic photography systemof claim 5, wherein an error of the distance differences is less than0.1 pixels.
 7. The stereoscopic photography system of claim 1, whereinthe pixels are positioned in a search interval of the pixel row.
 8. Thestereoscopic photography system of claim 1, wherein the synchronizationmodule provides a clock controlling signal for the first image sensorand the second image sensor.
 9. The stereoscopic photography system ofclaim 8, wherein the first data stream and the second data stream havesynchronized data line signals.
 10. The stereoscopic photography systemof claim 1, wherein the stereo matching module is implemented bysoftware, hardware, firmware, or a combination thereof.
 11. A method forperforming stereoscopic photography, comprising the steps of: capturinga first image and a second image respectively by a first image sensorand a second image sensor for generating a first data stream and asecond data stream; synchronizing the first data stream and the seconddata stream, and synchronizing auto-exposure and auto-white balanceparameters of the first image sensor and the second image sensor;combining the first data stream and the second data stream foroutputting a combined image data which comprises the first image and thesecond image; performing an image calibration and a horizontalrectification for the first image and the second image in the combinedimage data; and calculating a distance difference between a referencepixel of the first image and a corresponding pixel of the second image,wherein calculating the distance difference further comprises the stepof comparing costs of a plurality of pixels relative to the referencepixel in a pixel row of the second image in an interval manner.
 12. Themethod of claim 11, wherein the interval manner is an interval of apredetermined number of the pixels.
 13. The method of claim 12, whereinthe predetermined number is a positive integer.
 14. The method of claim11, wherein the costs are brightness differences.
 15. The method ofclaim 11, wherein calculating the distance difference further comprisesthe step of performing a sub-pixel interpolation for computing thedistance difference.
 16. The method of claim 15, wherein an error of thedistance differences is less than 0.1 pixels.
 17. The method of claim11, wherein the pixels are positioned in a search interval of the pixelrow.
 18. The method of claim 11, wherein the step of calculating thedistance difference is implemented by software, hardware, firmware, or acombination thereof.
 19. A stereo matching method of a stereoscopicphotography system, the stereoscopic photography system comprising afirst image sensor for capturing a first image and a second image sensorfor capturing a second image, the method comprising: calculating adistance difference between a reference pixel of the first image and acorresponding pixel of the second image, wherein calculating thedistance difference further comprises the step of comparing costs of aplurality of pixels relative to the reference pixel in a pixel row ofthe second image in an interval manner.
 20. The stereo matching methodof claim 19, wherein the interval manner is an interval of apredetermined number of the pixels, and the predetermined number is apositive integer.